What's the worst that could happen? - Clearing vacant lots, patching streets

[TheNiche]

I'm looking at an interestingly-situated parcel of land, but it is embedded within a completely overgrown section of other lots and decrepit shell/gravel/dirt streets which ultimately back up to the floodway. My question is, if I buy one lot and try to put something on it, patch the muddy sections of the old street with gravel just to gain reliable access, and then start clearing out the underbrush for the rest of the lots back toward the bayou, am I placing myself at any particular legal risk? (You know, short of coming upon the obligatory dumped body?)

Let's assume that I piss off the neighbors from on down the road, which is entirely possible if not likely. Could the City of Houston come after me for my guerilla street improvements? Is there any risk or downside to trespassing on a vacant lot, clearing it, and using it as an extension of my yard (with the ultimate intent of adverse possession)? Would the landowners have any claims against me if they figure out what I'm attempting to do? Is there any circumstance under which this could totally and completely backfire?