Crawler and sources
How this index is built
Heavy Vehicle Marketplace is a meta-search. It does not host the listings you see on the comparison pages. It records the headline facts about a listing, groups them so one model can be compared across sellers, states and sites, and then sends you to the site that is advertising the machine.
Our crawler
It identifies itself on every request as:
HeavyVehicleMarketplaceBot/1.0 (+https://heavyvehiclemarketplace.com/bot)
- It reads your robots.txt before every fetch and obeys the group that applies to it, including Crawl-delay. It never uses a browser User-Agent to get around a rule.
- It fetches no faster than one page every two seconds per site, and slower if your robots.txt asks for slower.
- A 403, a rate limit or a bot challenge is treated as a refusal. The source is dropped and the reason is published on this page rather than worked around.
- It stores headline facts only: make, model, year, asking price, hours or kilometres, state and suburb, the seller name, one thumbnail reference and the URL of your listing. It does not store your listing description and it does not copy your photo galleries.
- Listings that stop appearing at the source drop out of the index.
Sites currently indexed
Machines4u
www.machines4u.com.au · checked 2026-07-30
robots.txt allows /browse/ and /view/advert/ for User-agent: *, the site returns 200 to a named crawler UA, and every advert carries schema.org Product data.
Sites deliberately not indexed
These are the other major Australian heavy vehicle sales sites. Each one was checked against its live robots.txt and its live pages. They are left out for the reason given, and each would be included tomorrow through a feed or a partner arrangement.
Trade Trucks
www.tradetrucks.com.au · checked 2026-07-30
robots.txt is fully open and the 20,361 URL detail sitemap is served, but every /detail/ page answers a named crawler UA with a CloudFront 403. Getting facts out would mean defeating that block.
trucksales
www.trucksales.com.au · checked 2026-07-30
listing pages return a 403 JavaScript challenge to a named crawler UA, and robots.txt name-bans competing aggregators (trovitBot, CarsnipBot, ShopWiki).
Machinery Trader
www.machinerytrader.com.au · checked 2026-07-30
listing pages return a Cloudflare interstitial to a named crawler UA.
Gumtree
www.gumtree.com.au · checked 2026-07-30
listing pages return a 403 with an obfuscated JavaScript bot challenge to a named crawler UA.
Removing a listing or a site
Disallow the User-Agent above in your robots.txt and the crawler stops on its next run, with no further contact needed. If you would rather we indexed a feed you control, or you want a specific listing taken down, ask and it will be done.