You can buy simboxes at alibaba[0] which are routers with 12 to 64 simcards. Each simcard has it’s own port which you can use in the config of for instance playwright or scrapy. We use one in production for frontend testing and some scraping.
It works perfect but it’s not cheap and keeping track of simcards which are prepaid or subscriptions in combination with data usage is a bit of work.
Resetting a sim usually gives a new IP and our experience is that blocking an IP by a scraped party is rarely the case because of the shared nature of IPs in the mobile world.
Those are absolute overpriced crap. We scrape about 2T a month using mobile proxies, and testing those put us a in month-long nightmare of crashing, errors, connection drops, etc.
What other solutions do you suggest? My experience with 4G dongles is that they are really unstable (even buying the Proxidize hardware + software license).
At the end, solution that has worked better for me has been VPN + 5G phones. But it would be great to get rid of the batteries ...
Keep in mind that this doesn't need to be a Raspberry Pi and can be any computer. With some networking magic (and potentially network namespaces) you can also run multiple dongles & proxies on the same machine. A bit of hardware engineering will allow you to decouple the SIM card from the modems themselves, so SIMs can be kept at a central location and assigned to modems at will, potentially automatically depending on remaining credit/carrier blocks/rate-limiting/etc.
Have you tried probing for 4G provider(s?) IP renew limits and IP allocation space size?
Things I can think of that would be interesting to know:
1. How uniformly distributed are the IP addresses you're getting.
2. The size of the IP addresses allocation space, assuming a fixed geographic location (I imagine this can vary a lot between providers and regions)
3. How often can an IP be requested?
4. Is there some sort of rate limiting? If so, how does it manifest itself (Sticky IP? slow reconnect? Something else?)?
The distribution of IP addresses depends to a large extent on 4G provider. Some of them reuse IPs and you can get the same IP after you change network mode 4G > 3G > 4G.
The same applies to IP addresses allocation space.
You can request IP change as often as you want. Resetting network mode takes 5-10 seconds.
There is not rate limiting. From time to time reconnection fails but it has nothing to do with rate limiting. Sticky IP is an issue for some 4G providers but, again, it has nothing to do with rate limiting
Keep in mind that this behavior will be a significant outlier in ISP analytics - they will notice you eventually, just like they’ll notice 10 seemingly synchronized Huawei modems connecting to (and disconnecting from) the same base station.
That’s all to say - be careful using this for any nefarious activity. It’s not inconspicuous.
In Australia it's not possible [1] to get SIM cards without identification. So getting on the radar of the security folks at your telco could easily end with you never being able to use that telco again.
> Residential: provide IP addresses from Internet Service Providers (ISP) pool that are shared with other users
> Mobile: the best class of proxies for web scraping that is based on ephemeral IP addresses which are frequently exchanged with mobile network users who move between Base Transceiver Stations (BTS)
I can change my home IP address whenever I want... just change my router's MAC address and reboot the modem.
I'm not an ISP but I assume your previous IP addresses aren't released immediately so you could end up impacting the ISP's business by using up too many IP addresses.
If they notice, you might get banned. You could argue that you're not violating the terms of service but it wouldn't get you very far.
The modem can attach to the IMS APNs, however you'll need to do SIM-specific authentication which requires being able to send the right APDUs to the SIM. Some modems expose the SIM card in one way or another (either an AT command to send APDUs and get responses or as an actual USB smartcard reader) and it would be possible with those to do the authentication flow and then register onto the IMS server (which speaks SIP).
Alternatively, if you're happy with circuit-switched calls, some modems will expose the raw audio (as PCM data) as a separate serial port from which you read/write, and control the call flow using AT commands on the main serial port. Some modems need this to be first enabled via a poorly-documented AT command, and some might need authentication (security by obscurity) to do so which software such as DC Unlocker can break.
In addition to the EAP-AKA' authentication, the strange and highly opaque ways through which VoWiFi support is signalled and enabled might be an issue here too.
I've seen brand new Pixel devices (which fully support VoWifi) act like they have never heard of the feature, until a special cell broadcast SMS is received from the carrier, to turn the feature on.
It would be interesting to see if you could manually tick all the boxes and open up a tunnel to the ePDG etc without that happening, but given the number of moving parts, and the differences between each network's implementation, I'd be sceptical it would work reliably.
(I'm pretty sure I had a situation where individual batches of IMEIs were being whitelisted for WiFi calling on another network, where the switch appeared, but VoWiFi didn't work... Yet on the Nexus device in question, one bought from the carrier directly did work. Even when you manually flashed both devices with identical firmwares from Google directly, this remained the case - must have been an IMEI whitelist or similar).
Ah I see, so I did find someone that tried to do this here [1], but it seems like they were not even able to connect to the SIP server. Bummer, I wonder if someone else has had success.
It works perfect but it’s not cheap and keeping track of simcards which are prepaid or subscriptions in combination with data usage is a bit of work.
Resetting a sim usually gives a new IP and our experience is that blocking an IP by a scraped party is rarely the case because of the shared nature of IPs in the mobile world.
[0]https://a.aliexpress.com/_m0XnDLm