New York’s new cybersecurity rules for banks, insurance companies, and registered cryptocurrency groups, may leave small and medium-sized financial institutions struggling to comply.

The New York Department of Financial Services rules, which has a Sept. 4 compliance deadline, require companies to keep audit trails for online financial transactions and cybersecurity episodes. Companies also will need to maintain policies and procedures to secure financial apps used on mobile devices.

Many larger players, such as JPMorgan Chase & Co., The Bank of New York Mellon Corp., and Citigroup Inc., are used to following at least some of the rules already, attorneys said....