[Context and Motivation] Recent studies have highlighted transparency and explainability as important quality requirements of AI systems. However, there are still relatively few case studies that describe the current state of defining these quality requirements in practice. [Question] The goal of our study was to explore what kind of ethical guidelines organizations have defined to develop transparency and explainability of AI systems. We analyzed the ethical guidelines in 16 organizations representing different industries and public sector. [Results] In the ethical guidelines, the importance of transparency was highlighted by almost all of the organizations, and explainability was considered as an integral part of transparency. Building trust in AI systems was one of the key reasons for developing transparency and explainability. Customers and users were raised as the main target groups of the explanations. The organizations also mentioned developers, partners, and stakeholders as important target groups needing explanations. The ethical guidelines contained the following aspects of the AI system that should be explained: the purpose, role of AI, inputs, behavior, data utilized, outputs, and limitations. The guidelines also pointed out that transparency and explainability relate to several other quality requirements, such as understandability, privacy, security, traceability, and fairness. [Contribution] For researchers, this paper provides insights into what organizations consider important in the transparency and, in particular, explainability of AI systems. For practitioners, this study highlights the dimensions to be considered when developing explainable AI.