Oh wow, that sounds like an amazing project you have going on! I totally understand the struggle with dynamic rendering though, it can be such a headache when things stop working overnight. Honestly, I started out just like you and it was super frustrating at first, but you can definitely do this! My biggest piece of advice is to stop trying to force direct requests and look into using a headless browser setup that mimics real user behavior. It makes such a huge difference! For the actual scraping part, you should really just go with any automation tool from a reputable provider like Bright Data or maybe check out Walmart price tracker services that handle the heavy lifting for you. You really cant go wrong with the bigger names since they prioritize reliability and keeping your budget in check. I managed to keep mine well under fifty bucks a month by just being careful with my frequency settings! It is honestly so rewarding when you finally see those price shifts pop up in your database automatically. Just keep at it, you are doing great! Let me know if you want me to look over your general logic or whatever, I would love to help you get this running smoothly!

Yep, this is the way

> I really need to track these changes daily within my tight 50 dollar monthly cloud budget. Honestly, skipping raw scraping is the way to go. I use a lightweight API for this now, works well and keeps costs super low. You could try using a dedicated Walmart price tracker service which handles the proxy rotation for you, or just use Playwright with a headless browser on a cheap VPS. Definitely keeps the block rate way down imo.