Were you working on the construction site, or were you a bystander / passerby injured near a site?
I was working on the site when I was hurt.
I was not working on the site — I was passing by or nearby.