Cyberattacks: a Growing Threat to Marin Government

Observations 24

• OB1
  Email-based attacks succeed due to poor user behavior and can be greatly reduced by training to instill good user behavior.
• OB2
  The MIDAS platform does not prevent email-based breaches or filter for viruses.
• OB3
  To the Grand Jury’s knowledge, the county breaches and the $246,000 loss were never disclosed publicly, and the only municipal breach that became known to the general public was Sausalito’s. By not being sufficiently informed about the cybersecurity risks that exist in our cities and towns, the public may have a false Marin County Civil Grand Jury Page 7 of 18 Cyberattacks: A Growing Threat to Marin Government sense of security regarding effective government operations. Public transparency is essential so that Marin residents are aware of cybersecurity risks.
• OB4
  While the Sausalito and Novato breaches and one of the county breaches were reported to the FBI, the Grand Jury was unable to determine whether the other incidents were reported to federal law enforcement. The FBI recommends that government agency breaches be reported as a standard practice.9 In addition, when unauthorized disclosure of personal information occurs, California law requires the agency to notify all affected individuals, as Sausalito did in this case.10
• OB5
  Sausalito, according to interviews, responded appropriately to its breach. It not only notified the FBI, but provided identity theft protection resources to its employees, held a city council discussion on cybersecurity, and implemented a number of measures to strengthen its security, including mandatory employee training, technology for flagging external emails, and ongoing monitoring of its system by a cybersecurity consultant.
• OB6
  Partly as a result of the breaches to its systems, the county has acquired expert knowledge about techniques and strategies to prevent breaches and is in a strong position to share that expertise with cities, towns, and other agencies in Marin that may lack access to such practical knowledge. City and Town Officials Are Not Sufficiently Engaged in Combating Cyberattacks In municipalities across the United States, cybersecurity awareness and support for a stronger approach from elected representatives and top officials appear to be lacking. In 2016, the International City/County Management Association surveyed the chief information officers (CIOs) of U.S. county and city governments regarding cybersecurity issues. The survey asked the CIOs about the engagement of their top appointed or elected officials in cybersecurity risks and found, among other things, the following:11
• OB7
  Only 26 percent of the CIOs believed that elected council members were either moderately or exceptionally aware of cybersecurity issues.
• OB8
  Only 30 percent of the CIOs reported that elected council members provided either strong or full support for cybersecurity.
• OB9
  According to the CIOs, a very low percentage of elected and appointed officials felt they personally had a strong responsibility for cybersecurity. The survey results indicate that, while there is some awareness about cybersecurity risks, there is a lack of engagement by local elected officials in ensuring strong security. Marin is no exception 9 FBI, Law Enforcement Cyber Incident Reporting, accessed April 15, 2020, https://www.fbi.gov/file- repository/law-enforcement-cyber-incident-reporting.pdf/view. 10 California Civil Code § 1798.29, accessed April 15, 2020, https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=CIV&sectionNum=1798.29. 11 International City/County Management Association, Cybersecurity 2016 Survey, pp. 3, 12. Marin County Civil Grand Jury Page 8 of 18 Cyberattacks: A Growing Threat to Marin Government to the survey findings. As the above discussion of past attacks in Marin noted, city and town councils have not taken up the cause to raise public awareness or to combat cyber risks. In most cases, information security is an operational issue delegated to the town or city manager. In those municipalities suffering a breach, the Grand Jury found only one instance—Sausalito—where a city council directed its manager post-breach to provide the council with an assessment of its cybersecurity practices or measures that could be taken to improve security. While city and town managers in Marin are generally aware of the increasing number of cyberattacks, there appears to be a lack of action on the issue. In the course of its investigation, the Grand Jury heard repeated comments similar to the following:
• OB10
  Since we have a full backup, we are not too concerned about losing data (this, despite the fact that restoration of an untested backup might fail and an attack could cause loss of the current day’s data and an interruption in government operations lasting several days or more). There appears to be an overconfidence in the ability of a backup to enable a municipality to recover rapidly from an attack.
• OB11
  Our data is mostly a matter of public record anyway, so we are not too concerned about public disclosure (in fact, much of their data is confidential, including human resources data and information about pending litigation, not to mention information about private citizens that could be used in identity theft). There appears to be insufficient concern about the government’s need to protect important, confidential information. The Public Is Underinformed about Cybersecurity Threats to Our Government None of the breaches described above (other than the Sausalito attack) resulted in any public discussion by the governing boards of cyber threats or a demand from the board that the manager report to it regarding steps being taken to reduce those risks. The absence of a public discussion of these vulnerabilities is a missed opportunity to educate employees, residents, and local organizations about the cybersecurity risks faced by all. The Grand Jury heard two separate views on the wisdom of discussing these matters publicly. The first is that public disclosure would alert potential hackers that a jurisdiction is vulnerable to an attack. The second view is that, by disclosing and openly discussing the problem, coupled with taking strong action to improve network security, the jurisdiction makes clear its commitment to a high level of vigilance and security and reduces its attractiveness to potential hackers. While it would never be prudent to disclose in detail any technical vulnerabilities that led to a breach, the fact is that most attacks are launched when an employee clicks on a malicious email, and disclosing such an incident would not increase a municipality’s vulnerability but could serve to educate employees and residents of the importance of good user behavior. Unless disclosure would clearly create new security risks, the Grand Jury strongly favors public disclosure of these incidents. Marin County Civil Grand Jury Page 9 of 18 Cyberattacks: A Growing Threat to Marin Government Our Cities and Towns Should Adopt Best Practices to Improve Security A strategy followed by many smaller private and public organizations is to adopt “best practices” identified by IT professionals as a way of ensuring that they keep up with constantly changing risks.12 The Grand Jury investigated industry-standard best practices, as well as practices implemented successfully by various Marin agencies, and this report recommends that a number of them be implemented by all cities and towns in Marin. The National Institute of Standards and Technology has created its Cybersecurity Framework to assist governmental agencies and others with their security planning and practices. It identifies five key steps to planning and implementation: identify, protect, detect, respond, and recover.13 All public officials and managers should become familiar with its guidance and principles. Smaller cities and towns may believe that they cannot afford stronger security. However, the Grand Jury concluded that there are a number of inexpensive measures that every municipality should implement, if they have not done so already, that would materially strengthen their security. Employee training User behavior is at the center of cyber vulnerability and poses one of the greatest security challenges because it is difficult to change. Phishing attacks are initiated by email. They exploit employee behavior to gain network access. The more aware users are of hacking tactics, the better able they will be to avoid attacks—whether they are working in the office or at a remote location. The Grand Jury recommends regular, mandatory employee training to educate, motivate and, yes, scare, employees into following security practices. (One manager informed the jury that employees should feel “terrified” about what could happen in an attack.) One technique is to send fake emails to employees on a random basis to identify which employees have poor security discipline so that those employees can receive more training and more controlled network access. The jury recommends a service like this for all municipalities. Email Flagging and Filtering Malicious emails are often disguised to appear as if they came from within the organization, tricking the user into believing the email is from a colleague. To help counter this deception, the email system should place a visible “flag” on any email sent by someone from outside the organization. The County of Marin and several Marin cities and towns have already implemented such a system, but not all. Those that have done so report that the flag system has greatly improved user behavior. For a higher level of protection, the organization could implement a 12 Ekran System, “12 Best Cybersecurity Practices in 2020,” https://www.ekransystem.com/en/blog/best-cyber- security-practices; MetroStar Systems, “13 Cybersecurity Best Practices You Should Apply in 2020,” https://www.metrostarsystems.com/cyber-security/13-cybersecurity-best-practices-apply-2020/; ObserveIT, “10 Essential Cybersecurity Best Practices for 2019,” https://www.observeit.com/blog/10-essential-cybersecurity-best- practices-for-2019/. 13 National Institute of Standards and Technology, Framework for Improving Critical Infrastructure Cybersecurity, Version 1.1, April 16, 2018, https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.04162018.pdf. Marin County Civil Grand Jury Page 10 of 18 Cyberattacks: A Growing Threat to Marin Government system, as Sausalito has done, that will not deliver an email until the recipient verifies that the actual email address of the sender is the same as the purported address. All email systems should also have filters, sometimes called spam filters, that identify suspicious emails. Rather than letting these emails be delivered, the system “quarantines” them (or it may delete some emails entirely, depending on the security settings). The intended recipient receives a daily email listing all the quarantined emails and can then opt to have the emails he or she deems safe to be delivered. The rest are deleted. The Grand Jury recommends that all Marin cities and towns not only have such filters on their email systems, but also use the highest security settings available, consistent with operational needs. Password Management Strong, enforced password policies are essential to network security. If users create easy-to-guess or weak passwords, hackers can easily gain access. Password policies should require users to use complex passwords (using uppercase and lowercase letters, numbers, and special characters), to avoid sharing passwords or using the same password on multiple systems, and to change passwords periodically, at least every six months. With most systems, these policies can be enforced automatically by the system. User accounts also need to be managed tightly. When an employee leaves the organization, the account should be disabled immediately. In addition, many employees are given access to ancillary accounts, such as the municipality’s website, its social media accounts, its wifi network, and other cloud-based systems. The organization should create documented security procedures to inventory all of these other user accounts and ensure conformity with password policies. Organizations should be encouraged to use “password managers” where feasible. A password manager is a software program that performs like a vault to store all of your passwords and automatically log you in to a website where you have an account. By making password managers available to staff, the agency enables users to create very complex passwords that do not need to be memorized and to use different complex passwords on every system that they access. System administrators should also consider deploying “two-factor authentication” in certain cases. In addition to requiring the user to enter a password, this security feature requires the user to provide a second security credential before getting access. Most people have experienced this when they receive a text message containing a special code that must be entered before they can log on to a website. Two-factor authentication certainly should be used to access laptops and other mobile devices. It should also be required when accessing the system from outside the network. Data and System Backups Backups make a copy of the data on a computer or server to an alternative location to enable recovery from a data loss or a system lockup. Data should be backed up at least daily, although some systems allow data to be backed up throughout the day, which is better. Server backups should be made regularly to enable servers to be restored entirely from scratch to recover from ransomware attacks and similar outages. Marin County Civil Grand Jury Page 11 of 18 Cyberattacks: A Growing Threat to Marin Government While it is easy to set up a system for regular backups, the system should be tested regularly to confirm that the data can actually be restored. Backups are notorious for failing. Backups should be monitored for failure, and testing should be done at least monthly. The Grand Jury’s interviews revealed that city and town officials generally do not know whether their backup systems are ever tested. Other Best Practices There are a variety of other best practices that all cities and towns should evaluate for implementation, including these:
• OB12
  Management of mobile devices. Phones, tablets, laptops, and other mobile devices pose special risks because they are more susceptible to being lost or stolen. An agency should either prohibit the use of mobile devices to access government data or ensure that it has a platform to manage mobile devices. This system should include (1) enabling password management controls, (2) requiring two-factor authentication, (3) requiring use of a virtual private network, (4) encrypting all information stored on the mobile device, and (5) enabling “remote wipe” so that when a device is lost, its data can be deleted remotely.
• OB13
  Automated malware detection and removal. Antivirus software on the servers and personal computers can detect and remove malware before it does any damage.
• OB14
  Monitoring systems. Despite best efforts, most systems will end up being penetrated. It is important to have a monitoring system enabling the manager to see what is happening on the system and be alerted immediately when hackers have gained access.
• OB15
  Use of expert resources. Cyber threats are constantly evolving, and it is difficult for the average IT professional to stay current. It is critical to have access to an expert outside resource, especially when performing vulnerability assessments. Free resources such as the MS-ISAC alerts and newsletters can keep city and town managers (or their outside consultants) aware of new threats and risk-reduction techniques.
• OB16
  Firewalls. A firewall is a hardware device or software element that can block and filter outside access to a network. Firewalls should be up to date and deployed with security settings that are as strong as feasible, blocking, for example, all access from outside the United States.
• OB17
  Hardware and patching. Many attacks happen because older computer operating systems are no longer supported and cannot be patched with up-to-date software. It is common to replace computers every three to four years to minimize this problem. Grand Jury interviews revealed that many cities and towns lack any policy on how frequently they replace their equipment.
• OB18
  Documentation. All security measures and policies should be adequately documented and disseminated to ensure that (1) the policies and procedures are understood and capable of being followed, (2) users understand the expectations Marin County Civil Grand Jury Page 12 of 18 Cyberattacks: A Growing Threat to Marin Government placed on them, and (3) when employee turnover occurs, critical information about information security is not lost.
• OB19
  Vulnerability assessments. For organizations that can afford this extra step, a vulnerability assessment involves inventorying all systems, hardware, and software and assessing the points of vulnerability. A vulnerability report typically includes a list of recommended modifications. These assessments are usually performed every few years. Assessments can also include a “probe” element, where a deliberate attempt to gain unauthorized access to a system is made in order to educate users about vulnerabilities. Municipalities Should Work Together for Increased Security Forums for Information Sharing and Collaboration The Grand Jury’s investigation revealed that staff and elected officials in many Marin cities and towns are unaware that other jurisdictions in the county have been successfully attacked. Without this important information about breaches occurring among their peer group, city and town managers, as well as elected officials, are not alerted to the urgent need to reexamine their own security practices and to collaborate with their peers to improve the security of the entire group. More transparency and better collaborative approaches could help Marin’s smaller cities and towns become more sophisticated in their cybersecurity practices at a reasonable cost. Two existing groups that are well positioned to foster collaboration in this area are the Marin County Council of Mayors & Councilmembers (MCCMC) and the Marin Managers Association (MMA). One stated purpose of MCCMC is to promote cooperation and collaboration among Marin’s cities and towns “in the solution of mutual problems.”14 MCCMC has ad-hoc subcommittees devoted to such topics as disaster preparedness, homelessness, pension reform, and climate change, but they have no group devoted to cybersecurity. The Grand Jury’s investigation revealed that MCCMC has not had a focus on helping cities, towns, or other agencies improve their cybersecurity practices. By making cybersecurity a priority and creating a public forum for discussion of the issue, MCCMC could promote greater cybersecurity awareness not only among mayors and councilmembers, but also among the public, local businesses, and nonprofit organizations. MMA is composed of all of the town and city managers in the county, as well as the county administrator and the executive director of the Marin Municipal Water District. It serves as a forum for the managers not only to share their experiences and best practices for managing Marin’s cities and towns, but also to exchange ideas about how they might share services to lower costs and improve efficiency. The Grand Jury’s investigation revealed that MMA could do a better job of ensuring that experiences like the breaches described in this report are shared 14 “About,” Marin County Council of Mayors & Councilmembers, accessed April 15, 2020, http://www.mccmc.org/about/. Marin County Civil Grand Jury Page 13 of 18 Cyberattacks: A Growing Threat to Marin Government among its members, and that a higher priority is placed on cybersecurity in Marin’s cities and towns. Working in conjunction with the county’s chief information security officer, MMA could assist the cities and towns in distilling the above suggestions regarding best practices to a specific list for implementation. In addition, the county’s chief information security officer could start a special email list for city and town officials to keep them informed of cybersecurity alerts sent out by federal authorities, as well as provide regular email reminders to city and town staff to be prudent with external emails, attachments, and passwords. All of these efforts could be implemented at minimal cost. Shared Services Larger organizations can afford stronger security. For example, the county government has nearly 2,100 employees, more than 70 employees in its IT department, and a substantial IT budget. Marin’s two largest cities, San Rafael and Novato, also have substantial IT budgets and devote significant resources to cybersecurity. On the other hand, several of Marin’s smaller cities and towns do not have a full-time staff member devoted to IT management, using outside vendors instead. Marin’s cities and towns could turn to the Marin General Services Authority (MGSA) for assistance. MGSA is a joint powers authority formed for the purpose of administering shared programs among the county, cities, and towns.15 With a shared program, each participant generally contributes a fixed amount per year for MGSA to manage the program. In turn, MGSA generally contracts with an independent consultant to deliver services to the participating jurisdictions. For example, MGSA could establish a contract with an outside cybersecurity expert, who could then consult with individual cities and towns regarding their vulnerability and actions they could take to improve their security. Members could pay a base fee in exchange for a nominal service level, and then pay extra should they need more extensive consulting services. A shared cybersecurity program could be more effective than each city and town hiring its own consultant, because the MGSA consultant would acquire specific knowledge about the capabilities of the MIDAS wide area network and would not need to relearn those details on each assignment. Beyond cybersecurity, MGSA might also explore the creation of shared IT procurement standards for cities and towns. For example, every city and town needs a financial management system for its budgeting, fund accounting, and human resources needs. If all the cities and towns were to standardize on the same third-party software, they would be in a much better position to negotiate for lower prices and to create cross-jurisdiction user groups to enhance all users’ knowledge of how to use the system effectively. But if each city and town continues to act independently with regard to software selection and purchasing, efficiencies like this will not be possible. 15 “History and Overview,” Marin General Services Authority, accessed April 15, 2020, http://maringsa.org. Marin County Civil Grand Jury Page 14 of 18 Cyberattacks: A Growing Threat to Marin Government By moving toward a stronger culture of collaboration regarding IT needs, not just for cybersecurity, cities and towns would be able to enhance their performance while reducing their costs. MIDAS Enhancements Could Improve Security The county’s MIDAS wide area network has provided a strong and secure backbone for Marin’s municipalities for the past 25 years. With its firewalls and redundant, secure connection to the internet, it provides a good first line of defense against cyber criminals. However, as previously discussed, attacks that use fake emails as their entry point are not stopped by MIDAS, and MIDAS does not currently provide malware filtering or antivirus protection. In addition, the Grand Jury heard concerns that MIDAS is too costly and the internet speeds are too slow, which could result in some cities and towns deciding in the future to opt out of the system. This might weaken the security they currently enjoy. Given the county’s strong Information Services and Technology Department and its many years of experience with the MIDAS system, the county is well positioned to provide additional support and resources to Marin’s cities and towns regarding cybersecurity. In 2020, the county is performing a review of the MIDAS system for possible modifications, enhancements, and cost reduction. Modernizing and enhancing MIDAS could provide even more security, which would create a strong motivation for cities and towns to continue using the system or even rely on MIDAS more. Enhancements could include the following:
• OB20
  Web filtering, where particular websites, especially those known to host malware, could be blocked automatically, or “blacklisted”
• OB21
  Geo-blocking to block websites from certain countries or regions
• OB22
  Email filtering to prevent known malware from getting through
• OB23
  Real-time monitoring dashboards for better management capabilities
• OB24
  Disaster recovery features While these enhancements would undoubtedly come at some cost, it may be possible to make them elective for those cities and towns that believe the costs are justified.

Agency Responses 132

Government agencies' official responses to this report's findings and recommendations. Click on a response to see the structured breakdown.