A valid transmutation must be in the correct form and must not violate the rules regarding the fiduciary relationship between spouses.
To unpack that statement a bit, we first look to the "express declaration" requirement as alluded to California Family Code Section 852, which states the following:
(a) A transmutation of real or personal property is not valid unless made in writing by an express declaration that is made, joined in, consented to, or accepted by the spouse whose interest in the property is adversely affected.
(b) A transmutation of real property is not effective as to third parties without notice thereof unless recorded.
(c) This section does not apply to a gift between the spouses of clothing, wearing apparel, jewelry, or other tangible articles of a personal nature that is used solely or principally by the spouse to whom the gift is made and that is not substantial in value taking into account the circumstances of the marriage.
(d) Nothing in this section affects the law governing characterization of property in which separate property and community property are commingled or otherwise combined.
(e) This section does not apply to or affect a transmutation of property made before January 1, 1985, and the law that would otherwise be applicable to that transmutation shall continue to apply.
There's a fair amount of uncertainty as to how a court will interpret the "express declaration" used in a transmutation agreement between spouses. Because of this, practitioners often advise incorporating the word "transmute" or "transmutation" into the agreement (or some other wording that indicates the parties understand that the character of the property is being changed).
The statute above also clearly indicates that any such agreement must be in writing (with some exceptions as a result of case law) and the person who is negatively affected must accept or agree to the transmutation.
Though not directly mentioned in the statute above, a transmutation of property will not be upheld if a court determines that one spouse unduly influenced the other spouse. As a starting point, under California law, spouses are presumed to be in a confidential relationship with each other.
Section 1100(e) of the California Family Code states:
Each spouse shall act with respect to the other spouse in the management and control of the community assets and liabilities in accordance with the general rules governing fiduciary relationships which control the actions of persons having relationships of personal confidence as specified in Section 721, until such time as the assets and liabilities have been divided by the parties or by a court. This duty includes the obligation to make full disclosure to the other spouse of all material facts and information regarding the existence, characterization, and valuation of all assets in which the community has or may have an interest and debts for which the community is or may be liable, and to provide equal access to all information, records, and books that pertain to the value and character of those assets and debts, upon request.
If one spouse gains an advantage as a result of the transmutation, there's a rebuttable presumption that the transmutation was a result of undue influence. Once the presumption exists, it is up to the spouse with the advantage to rebut it. If he or she is unable to rebut the presumption, then the transmutation will be considered invalid.
Two areas that are not clear under the law involve (a) the degree of advantage one spouse must have before the presumption is invoked and (b) whether this fiduciary duty can be waived. Both of these are beyond the scope of this post, but should be noted when dealing with the subject of transmutation agreements.