Thinking this over, I think that indeed it could work given that context is clear.

Assume the following dialogue..

- I'm in a hurry to get to my house 

- Why ?

- Because I need to work 

- Didn't you work at the office ?

- HIja', 'a juHDaq jIvum je.

I don't think, that there would be anything confusing in such a reply, or anything controversial in using the {je} "too" this way.

~ mayqel qunen'oS