Senior Network Operations Engineer

Madison Square Garden Entertainment Corp. (MSG Entertainment) is a leader in live entertainment experiences. The Company presents or hosts a broad array of events in its diverse collection of venues: New York’s Madison Square Garden, Hulu Theater at Madison Square Garden, Radio City Music Hall and Beacon Theatre; and The Chicago Theatre. MSG Entertainment is also building a new state-of-the-art venue in Las Vegas, MSG Sphere at The Venetian, and has announced plans to build a second MSG Sphere in London, pending necessary approvals. In addition, the Company features the original production – the Christmas Spectacular Starring the Radio City Rockettes – and through Boston Calling Events, produces the Boston Calling Music Festival. Also under the MSG Entertainment umbrella is Tao Group Hospitality, with entertainment dining and nightlife brands including Tao, Marquee, Lavo, Avenue, Beauty & Essex and Cathédrale. More information is available at www.msgentertainment.com.  

The Sr. Network Operations Engineer will work with design and engineering teams to have an all-round understanding of networks and systems. The Senior Network Operations Engineer will be responsible for resolving complex issues that involve analyzing packets and troubleshooting networks, supporting and maintaining highly specialized rendering & VFX workflows. This role is the crucial point of contact between the network operations team and Studio Operations to ensure optimization of traffic/workloads.  The engineer must work towards improving the quality of technology services and optimizing performance.

The Sr. Network Operations Engineer will offer Level 2 and Level 3 Network Support that includes operation and change management, fulfils the request for Network Devices such as Routers, Firewalls, WAN Optimization. Senior Network Operations Engineer will examine, evaluate and monitor the incoming network change requests.  Resource should be well versed on the impact of technical decisions against studio/broadcast critical systems 

This position will also mentor team members in an effort to develop excellent enterprise networking skills.  

• The Sr. Network Operations Engineer supports the network and network connected equipment, identify and repair the internal and external communications of network/server/system. The Engineer executes fault isolation and diagnostic functions to ensure the execution of corrective actions on a real-time basis. They act as a technical specialist for troubleshooting and incident management during service interruptions. 

• The Sr. Network Operations Engineer is responsible for documenting LAN/WAN drawings, equipment lists accurately. They are responsible for the technical documentation of the network changes. The Sr. Network Operations Engineer conducts a periodic review of upgraded system firmware and application software. They will manage and administer the production IP communications equipment. 

• The Sr. Network Operations Engineer is responsible for creating production support availability, outages, Runbooks, Process documentation and status reports. They work in coordination with operational support teams to recognize and record monitoring and measurement requirements while working within a multi-location enterprise. 

• The Sr. Network Operations Engineer provides administration and reporting services in the areas of incident and problem management 

• This job involves tackling various network issues and contributing to the smooth functioning of the organization. The engineer will interact with the technical support, managed support, and service provisioning teams both internal and external. The professional will offer first-line network surveillance and management support. They are responsible for high-level network operations. 

• The Sr. Network Operations Engineer will offer technical assistance in regard to network design, implementation, and integration of network changes. He/She will help in designing and implementing analytical research, design development and other assignments as required. He/She performs various complex software development tasks. They ensure delivery of contract capabilities using industry standards and repeatable processes. 

• Designs and implements highly resilient and failure proof network solutions with the ability to withstand system failures, faulty code and other infrastructure impairments – thinks with a global mindset and possess a “never fail” mentality when implementing solutions. 

• Designs virtual network services in support of business applications and communication platforms – experience designing quality of service and micro-services across the network. 

• Create a program to visualize the performance of the network, monitor for health and report on problem areas 

• Works with Network Engineering to design networks for automation and intelligence – embraces the opportunities for machine learning and other advanced intelligent platforms to drive outcomes 

• Works closely with the Director of Studio Infrastructure to create capital and operational budgets that pertain to the equipment and services in use within the network infrastructure and Venue Technology Teams 

• Establishes testing strategy, standards and/or objectives in alignment with company enterprise IT objectives 

• Ensure stable and reliable ISP network services (e.g. access products, service networks and core network)  

• Day-to-day network performance monitoring  

• Network troubleshooting and fault analysis; hardware troubleshooting and repair  

• Trouble ticket generation and response; carrier interface and escalations  

• Deployment and maintenance of network monitoring, analysis and reporting tools  

• Application of configuration changes via our change control systems and processes   

• Installation of network hardware and software 

• Participate in a 24x7 call-out rotation 

QUALIFICATIONS:   

• 10+ years of enterprise network operations experience – applicant must be “hands on keyboard” with the ability to technically troubleshoot network related outages 

• Previous experience at an enterprise level company supporting complex networks with absolute fault tolerance in mind (Media & Entertainment experience preferred) 

• Proven hands-on experience deploying various tool sets which support monitoring and alerting across diverse platforms – has integration experience working with event correlation platforms and ticketing systems such as ServiceNow 

• 3-5 years of vendor management experience is required – must have managed complex vendor engagements with large organizations such as Cisco, Palo Alto, F5 and others 

• 3-5 years’ experience supporting large scale content distribution and processing workloads 

• At least 4 years of working with large scale infrastructure platforms including compute, storage, data centers and automation 

• Experience with advanced network operations utilizing various product platforms such as Cisco, F5, Citrix, Palo Alto and NetScout. 

• 3 years supporting and troubleshooting network connectivity to public cloud services providers specifically AWS and Azure  

• Experience working with broadcast networks and streaming technology – supported networks which feed content and data to social media sites and other digital properties 

• Must be fluent in all aspect of network security and security elements – supported networks with proper controls and governance from conception to execution; TPN implementation/management experience preferred  

• Bachelor’s degree in engineering, computer science or computer engineering is preferred or equivalent experience 

SUPERVISION:  

• Role is an individual contributor role but may mentor team members and lead Operational initiatives 

SPECIAL REQUIREMENTS OF THE JOB:  

• Candidate selected may have to travel several times throughout the year to other business locations 

• Advanced Cisco and/or Palo Alto training and certifications highly desirable