I do not think I would do that as the earthworms that you get from your yard are not run through the process of cleaning and disinfection that the ones are that you buy for feeding to fish. I would be afraid of contamination and/or disease processes being introduced into your tank. If you want to feed your fish actual food from a store get a small piece of liver from the grocery store or even some chicken livers and cut very small pieces of raw liver and allow the betta to have a couple small (half of his eye sized) pieces of it a couple times a week or so. But the bettas MAIN diet needs to be his bloodworms or pellets that are made for his consumption and not many of those. Remember that his stomach is the size of his eye and it is not to be overstuffed. Overfeeding of bettas, although done with the best of intentions, is one of the main causes of their death.

I have honestly had emergencies come up where I was not able to arrange for my fish to taken care of and been gone for 5 days with the fish getting no food for that length of time and they are fine. They really require very little food and can find microrganisms in the tank that we cannot even see to keep them fed for a time. I will say that I had some very clean tanks when I returned! They were very happy to see me but I only gave them enough to feed them one small meal. Discipline when feeding is so important in keeping your fish healthy.

You mentioned feeding your betta flakes. Their mouths are not really made for flakes but if you note that he is eating them then okay. They get along with pellets or better yet bloodworms or daphnia or some other form of food much easier. The brand name that I usually use simply because I have never had any difficulty with the size and/or quality of it is Hikari for the bloodworms or daphnia and Atison's by Ocean Nutrition for the pellets. Some others that work really well are:

Hikari Betta Bio-gold pellets
Atison's Betta Formula Pellets (bettas over 6 months old)
Atison's Betta Pro Pellets (bettas under 6 months old)
Hikari freeze dried bloodworms
Hikari freeze dried daphnia

I always soak the pellets for a few (up to 5) minutes as they swell in the bettas tummy and can be deceiving on how many should be fed and can get them overfull if you let them swell INSIDE the fish. The daphnia and bloodworms do not swell but you still need to control how many of them are fed at one time.

Rose