Let's get one thing out of the way - I am perfectly well aware of the reasoning behind the decision based on the intended use of Optional in APIs. I'm not going to challenge the decision directly - I am going to challenge the reasoning which has several fundamental flaws.
The first challenge is on the basis of the "Principle of Least Surprise". Everywhere I look on the Internet on this topic, going all the way back to the early pre-releases, developers were caught out and surprised by this decision. Developers are still being caught out and surprised by this and, I am willing to bet, will continue to be caught out by this for years to come. I have complete sympathy with them. The precursors of Optional including the Nullables tended to serialize, Optional is a carrier of other types which are often Serializable and Optional tends to work well in the streams coming out of Collections which are themselves serialization friendly.
The second challenge is in terms of the attitude behind the decision. I perceive the attitude as being very "Ivory Tower". In my head at least the reasoning was that "We, the great and good of Java, can only conceive of one way in which Optional should be used: Therefore we will do all that we can to prevent the proles from going against our superior understanding." If that was not the reasoning then it must have been something close... For some reason I hear "Inconceivable!", "You keep using that word, I do not think it means what you think it means."
My third challenge is on the basis of good API design. Good API design is contextual, it needs to be appropriate for the use case. Core language APIs are a special case in that they are should provide the most expressive, flexible and robust semantics as they are a major part of the toolkit with which you build everything else. Before something gets added to a core language API a key question should be asked: "Is this capable of being adapted to many useful use cases and being used in unexpected and novel ways that will aid the evolution of the language and its community?" The fact that Optional was envisaged for only one key use case and deliberately excluded from many others is a strong argument that it should never have been included in the core APIs in the first case.
What really saddens me is that there is a lot of good clear thinking resulting in a set of very flexible, robust and expressive additions to the core Java language in Java 8. Unfortunately Optional is not one of them when it so nearly could have been.
